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ABSTRACT 

Previous research into the behavior of students while 
learning to program by automatically recording their actions has revealed 
that such recordings contain a wealth of information that can be collected 
together into a diagnostic tool that can support students* learning. The 
first step has been to construct the Coach, a software component that can be 
invoked on demand to provide a variety of support based on students* previous 
experiences. The Coach has undergone an initial stage of usability and 
usefulness testing to determine its effectiveness in practice. This paper 
describes the design of the Coach and reports on a small-scale experiment to 
assess its effectiveness with two groups of students. It was found that 
students did indeed turn to the Coach for help and that the control group 
also searched for help, but had to get it elsewhere. The paper also reports 
on other differences in behavior between the two groups. Includes seven 
figures and two tables. (Contains 17 references.) (Author) 
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Abstract 



Previous research into the behaviour of students while learning to program by automatically recording their actions has 
revealed that such recordings contains a wealth of information that can be collected together into a diagnostic tool that 
can support students’ learning. The first step has been to construct the Coach, a software component that can be invoked 
on demand to provide a variety of support based on students’ previous experiences. The Coach has undergone an initial 
stage of usability and usefulness testing to determine its effectiveness in practice. This paper describes the design of the 
Coach and reports on a small -scale experiment to assess its effectiveness with two groups of students. It was found that 
students did indeed turn to the Coach for help and that the control group also searched for help, but had to get it 
elsewhere. The paper also reports on other differences in behaviour between the two groups. 



Introduction 

Students on the Open University introductory course in computing M206, Computing: An Object Oriented Approach 
(M206 2000), are taught the 00 paradigm using Smalltalk. The students, of which there are approximately 5000 per 
presentation, are taught at a distance. The teaching is supported by a series of practical activities performed with the 
LeamingWorks system (Goldberg, Abell et al. 1997) . The LeamingWorks environment consists of over 30 
LeamingBooks, each one of which contains a set of practical activities. The structure of a LeamingBook is based on the 
paradigm of a book in that the student can read through some pages of material that describe practical activities which 
have to be carried out in other pages of the book (Woodman, Griffiths et al. 1999). The early programming exercises 
ask students to interact with a series of micro -worlds in which they observe the effects of small portions of Smalltalk 
code; later exercises ask the students to construct their own code. 



The left-hand side of Figure 1 shows the contents list of LeamingBook 06, LB -06. All LeamingBooks are divided into a 
sequence of sessions. A session is designed to be studied in one continuous interaction with the computer. Each session 
comprises a sequence of practicals, and each practical has an associated discussion page. Thus, students are encouraged 
to attempt a practical and then examine the discussion where the outcomes of the practical are examined. 
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Figure 1 The interface of a typical LeamingBook 

On the right-hand side of Figure 1 is a page from LB-06 shown detached from the LeamingBook containing the 
Amphibians micro -world. This micro-world is intended to support the learning of message sending. The Amphibians 
page also contains an evaluation text area in which students can enter small sections of Smalltalk code for the system to 




2 



BEST COPY AVAEIABILE 



evaluate. In later LeamingBooks, the idea of a workspace is introduced where students can enter significantly larger 
pieces of code and investigate their execution (see Figure 2), 

Given this novel approach to the teaching of programming, we were interested to know how effective it vould be. 
Therefore, we set up a significant research project (Thomas, Macgregor et al. 1998) to investigate how students learn to 
program in this environment. Our first step was to develop a student Observatory - an electronic system for recording 
the actions that students take when interacting with LeamingBooks (AESOP 2001), An analysis of the recordings 
(Thomas and Paine 2000) showed a variety of student behaviours, particularly when dealing with errors (Logan and 
Thomas 2001). Therefore, we set about adapting the observatory software to provide additional support for error 
message comprehension and error correction - the Coach (Paine 2001) This approach opens up the question of how 
effective the Coach would be. There are two aspects to this question. First, how easy would students find the system to 
use and second, how useful would students find the system? The effectiveness of the Coach would be ascertained by 
examining the extent to which students used it to solve problems. In this paper we shall discuss the effectiveness of the 
Coach; the usability issues are discussed in (Thomas, Paine et al. 2000) . 



The Coach 

Whenever the LeamingWorks system detects an error in an item of Smalltalk code, it issues an error report. Figures 2 
and 3 show two kinds of report. In Figure 2, the error report is shown highlighted. To obtain help with this error the 
student can invoke the Coach by clicking on the appropriate button. In Figure 3, the report appears in a modal dialog 
box that would normally be cleared by clicking on the OK button. However, in the modified system, the student has the 
option of invoking the Coach. The result of clicking on the Coach button is a new window similar to the one shown in 
Figure 4. 




Objects of class Frog do not know 
how to respond to the message up 





OKi 1 


1 


1 Coach 1 



Figure 2 The LeamingBook interface 



Figure 3 A typical error report 



The prototype Coach window has two main areas. At the top of the window is a tabbed area labelled ‘Actions’ which 
enables the student to scroll through a history of their actions. This uses the Observatory’s recording of the student’s 
actions throughout the LeamingBook. The error report is shown highlighted and, in Figure 4, is a textual representation 
of the contents of a dialog box similar to the one shown in Figure 3 (the DIALOG is the error report and the CHOICES 
are the buttons that appear in the box). The second tabbed area that occupies most of the window shows a series of 
hints, the first of which is headed MEANING and contains an expanded explanation of the error that has been detected. 
The remaining items on the Hints page are possible reasons, given contextually, for the occurrence of the error. The 
hints contain hyperlinks to the Glossary containing definitions of the terms used in the descriptions. The remaining tabs 
give access to possibly useful materials such as the main teaching texts (Chapters), links to related web sites (Links) and 
a graphical Smalltalk syntax analyser (Precedence). 
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MEANING: The identifier "red" is not known to the LeamingWorks 
system. You must declare ftell the system about) each new identifier you 
want to use. 


T 


If you really do want a variable called "red". Create it, that is, declare it to 
the system. 




"red" may need the initial letter capitalized, there is a global variable called 
"Red" 




You may have misspelled or mistyped the identifier "red". You might like to 
try one of the following variable names instead (note some of these 
possibilities may not be very sensible suggestions for the problem in hand); 
"Red" "True" 
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Figure 4 The Coach interface 

Effectiveness experiment 

In a small experiment designed primarily to investigate usability issues, we took the opportunity to study the 
effectiveness of the Coach, 14 student volunteers were provided with two additional LeamingBooks, each containing a 
small number of practical exercises related to the work of LeamingBooks 09 and 10. The practicals asked the student to 
evaluate a number of simple expressions, each of which resulted in an error, and to attempt to rectify the errors. Those 
with the Coach were told of its purpose, but were not required to use it to solve the problems. 

The experiment used an independent samples design to compare the actions of students who used the Coach with those 
who did not (Siemer and Angelides 1998; Budgen and Pohthong 1999). Students were divided into two groups, 
providing us with 2 conditions. In condition 1, students were provided with the additional LeamingBooks, the AESOP 
Recorder and Coach software. In condition 2, students were given the additional LeamingBooks and the AESOP 
Recorder software (i.e., they did not receive the Coach software). Students were assigned to a condition on the basis of 
a pre -experiment questionnaire aimed at controlling variables related to gender and age. This resulted in 8 students 
being assigned to condition 1 group and 6 to condition 2 group. Having completed the test exercises, the students e- 
mailed their recording to us for analysis. 

Post-Questionnaire 

Once they had finished the tests, all students were asked to complete a questionnaire designed to assess usability and 
usefulness issues. When asked to rate the ease of completing the tests on a scale of 1 (difficult) to 5 (easy), on average 
students without the Coach rated the practicals as easier (4.4) than students with the Coach (3.75). Generally, students 
who used the Coach found the amount of information on its interface slightly distracting. When asked about the 
likelihood of using the Coach in other LeamingBooks on a scale of 1 (unlikely) to 5 (likely), on average they said 3.25. 

The following comment from one student summarises experiences with the Coach: “The information in the Coach was 
useful, although it was a little difficult to home in on the appropriate comments for the problems I had. When I went 
back to using LeamingWorks I found myself looking for it in my next LB and on a couple of occasions wishing it was 
there. Sometimes the Smalltalk error messages are difficult to interpret and I think the Coach would help.” 



Analyses of LB Test-09 Coach Recordings 

Of the 8 students with the Coach, 7 used it ‘for real’. One student did not open the Coach at any point during either test 
LeamingBook. All 8 students attempted all of the practicals. Figure 5 shows the percentage of students who used the 
Coach on each practical activity: it only includes those students who used the Coach to help them attempt to solve the 
practical i.e. not those who simply opened the Coach to look at it. It shows that, for each practical, some students felt 
the need to use the Coach. 
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Figure 5 The percentage of students who used the Coach in each practical 

With the exception of 1 student in one practical, all students with the Coach who solved the problem in a practical did 
so without looking at the discussion pages. However those students without the Coach who solved a practical 
sometimes looked at the discussion page before the solving the problem, as shown in Table 1. A few students without 
the Coach software also looked at the discussion page but did not solve the practical. We concluded that some students 
in each group needed additional support to solve the problems. 





No. of students who: 


Practical 


Solved practical 


Looked at Discussion page 


1 


6 


1 


2 


6 


0 


3 


5 


1 


5 


5 


2 


8 


6 


2 


10 


6 


1 


11 


6 


5 


12 


5 


0 



Table 1 : Students without the Coach who looked at the discussion before solving each practical 

Figure 6 shows the percentage of students who correctly solved the problems that had a single specific solution. There 
were students in each group who failed to solve some problems. In three out of these four cases, more students without 
the Coach succeeded in solving the problems. Nevertheless, in practical 1 , 8 and 1 1 , some students without the Coach 
tried the practical only after having read the discussion, so one might conjecture that not all students in this group 
would have successfully completed these exercises. However, the results of the post-questionnaire indicates that the 
group with the Coach were weaker. This is confirmed by the number of student errors (other than those mandated by the 
practical activities) in which those without the Coach made on average 18.83 errors each, whereas those with the Coach 
made 23 errors each on average. 




Figure 6 The percentage of students with and without the Coach who solved each problem 



A summary of the data extracted from the recordings for LB Test-09 from students with and without the Coach software 
is shown in Table 2. 





Average number of: 




Time spent 


Open 


Close 


Hyperlinks 


Evaluations 


Dialogs / Notifiers 


With 


48 mins 


2.63 


9.75 


39.25 


35.75 


35.00 


Without 


20 mins 


1.33 


4.50 


22.50 


32.83 


30.83 



Table 2 A summary of LB Test-09 recordings. 

From Table 2 it can be seen that students with the Coach spent, on average, over twice as long in LeamingBook Test-09 
than students without the Coach. 

Students with the Coach appear to do LB Test-09 in more sittings that student without the Coach (shown by the higher 
number of ‘Open’ events). Students with the Coach also on average accessed more hyperlinks than students without the 
Coach. This is to be expected as students with the Coach have access to the Coach Links, Chapters and Glossary pages 
which all contain a number of hyperlinks. 

We analysed the results of the second test (LB Test- 10) in a similar way. The difference between the two groups of 
students was less marked particularly in the amount of time spent solving the problems. This is to be expected as 
students get used to using the Coach software. This gives us confidence that using the Coach need not be a significant 
overhead, especially when it is clear that some students look for additional support to solve some problems. 

Future work 

We have implemented a revised Coach interface and slightly amended the two test LeamingBooks so that we can repeat 
this experiment during the 2002 presentation of the course but with a much larger sample of students. 

The next major step in the development process is to utilize the data contained in the student recordings to improve the 
feedback given by the Coach based on actual student experiences. Figure 7 shows the architecture of the system as 
envisaged. The basic idea is that the Coach obtains its data from a database on a student’s machine (downloaded with 
the Coach). The local database is augmented with additional data based on the student’s interactions with the 
LeamingBooks (using the Events Analszer component). The recordings are sent to a central recordings database as with 
the present system. The recordings will contain all the events that occurred in each LeamingBook, including those 
related to the use of the Coach since it is a LeamingBook, too. The complete set of all students’ recordings are analysed 
to update a central database of Coach data. Students will be able to update their local database with revised Coach data 
as and when they wish. In this way we hope to be able to adapt the Coach in the light of all students’ experiences. 




Figure 7 The Coach System 

A third avenue for exploration is to follow up on a discussion of models of intelligent tutoring in(Gertner, A. & 
VanLehn, K (2000)) by using the Coach to provide a model of how a given problem should be solved. This will be 
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based on work in another area [Thomas, 2001] for which we are developing mechanisms for specifying tasks and 
identifying attempts at solving them. We want to see how these mechanisms might be used to create a model of how a 
problem is to be solved and to detect student attempts at the solution. As the student tries to solve the problem, his/her 
actions are compared to those that the model would make. If the student’s actions diverge sufficiently from the model, 
the Coach would offer the student some advice or feedback. 



Conclusions 

Overall, the majority of students who used the Coach found it useful. However, the prototype interface was found to be 
distracting and students found it difficult to home in on the appropriate hint. We obtained useful feedback on the 
interface and have simplified it. On balance, the Coach seems to offer a beneficial tool that some students found 
attractive. 

We believe that the Coach is effective and that it is worth investing further effort to improve it. In particular, we can see 
ways of adapting the Coach to individual student needs. We also believe that it will be possble to provide further help 
through the idea of a model solution and comparing it with student attempts at solving programming problems. 
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